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Introduction 

This module is part of a collection of modules for a class project on matrix 
completion techniques for the sensor network localization problem done for 
the Fall, 2009 offering of Prof. Baraniuk's ELEC 301 course at Rice 
University. 


Introduction 


Sensor network localization refers to the problem of trying to reconstruct 
the shape of a network of sensors — that is, the positions of each sensor 
relative to all the others — from information about the pairwise distances 
between them. If all of the pairwise distances are known exactly, then the 
shape of the network may be recovered via a technique called 
multidimensional scaling (MDS) [link]. Of more practical interest is the 
case in which many — even most — of the distances are unknown and in 
which the known distance measurements have been corrupted with noise. 
Determining the shape of the network under these conditions is still an open 
problem. Over the years, researchers have come up with a variety of 
different approaches for tackling this problem, with some of the most recent 
ones being based on graph rigidty theory, such as those in [link] and [link]; 
however, for our project, we decided to examine this problem from a 
fundamentally different tack. Instead, we approach the problem using 
methods from the brand new field of matrix completion, which is concerned 
with “filling in the gaps" in a matrix for which not all of the entries may be 
known. 


The remainder of this collection is divided as follows. In the next section, 
we provide an overview of the most recent work in matrix completion for 
those who may not be familiar with this very new field. After that, we 
discuss the procedures we used to conduct our investigation. Finally, we 
examine the results of our simulations and present our conclusions. 


Matrix Completion: An Overview 

This module is part of a collection of modules for a class project on matrix 
completion techniques for the sensor network localization problem done for 
the Fall, 2009 offering of Prof. Baraniuk's ELEC 301 course at Rice 
University. 


Overview of Matrix Completion 


The fundamental question that the new and emerging field of matrix 
completion seeks to answer is this: Given a matrix with some of its entries 
missing, is it possible to determine what those entries should be? 
Answering this question has an enormous number of potential practical 
applications. To be more concrete, consider the problem of collaborative 
filtering, of which perhaps the most famous example is the Netflix problem 
[link]. The Netflix problem asks how one may be able to predict how an 
individual would rate movies he or she has not seen based on the ratings 
that individual has made in the past and on the ratings of other individuals 
stored in the database. This can be cast as a matrix completion problem in 
which each row of the matrix corresponds to a particular user, each column 
to a movie, and each entry a rating that the user of that entry's row has given 
to the movie in that entry's column. Because there is a large number of users 
and movies and because each user has probably seen relatively few of the 
available movies, there are a large number of entries missing. The idea is to 
somehow fill in the missing entries and thereby determine how every user 
would rate every movie available. For more examples of potential uses of 
matrix completion, see the introduction of [link]. 


In general, matrix recovery is an impossible task because the unknown 
entries really could be anything; however, if one makes a few reasonable 
assumptions about the original matrix underlying the one being completed, 
then the matrix can indeed be reconstructed and often from a surprisingly 
low number of entries. More precisely, in their May, 2008 paper Exact 
Matrix Completion via Convex Optimization, matrix completion pioneers 
Emmanuel J. Candés and Benjamin Recht offer the following definitions 
[link]: 


Definition: Let U be a subspace of IR” of dimension R, and let Py be the 
operator that projects orthogonally onto U. The coherence (U) of U is 
defined by 

Equation: 


n 
p(U) = — max || Pye; ||’, 


T 1l<i<n 


where e; is the standard basis vector with a 1 in the i*» coordinate and all 
other coordinates are zero. 


Definition: Let A be an m-by-n matrix of rank r with singular value 

* 
decomposition ee O;,U;V,, and denote its column and row spaces by U 
and V, respectively. A is said to be (jug, f41)-incoherent if 


1. There exists j1g > O such that max (u(U), u(V)) < po. 
2. There exists 4; > 0 such that all entires of the m-by-n matrix 


* 
So he1 UKY;, are less than or equal to 4; ,/— in magnitude. 


r 
mn 
Qualitatively, this definition means that the singular vectors of a (j19, /1)- 
incoherent matrix aren't too “spiky" and don't do anything “wild.” 


In the same paper, Candés and Recht go on to show that if A is an m-by-n 
(0, /41)-incoherent matrix that has rank r << N =max (m,n), then A can 
be recovered with high probability from a uniform sampling of M of its 
entries, where M > O(N 127 log N ) [link]. This result was later 
strengthened to M > O(.Nr max (r, log N)) by Keshavan, Montanari, 
and Oh in [link]. These results, coupled with the fact that many matrices 
that one encounters in practice both satisfy the incoherence property and are 
of low rank means that matrix completion has some serious potential for 
use in practical applications. 


Once one knows that matrix completion can be done, the next question is 
how to go about doing it. There are a variety of different matrix completion 
algorithms available. Candés et al. have developed a method that they call 
Singular Value Thresholding (SVT), which attempts to complete the matrix 


by solving the following optimization problem [link]: Find a matrix X of 
that minimizes || X || subject to the condition that the entries of X be 
equal to those entries of the matrix A to be completed for which we know 
the value. Here, || X || is the nuclear norm of X, defined to be the sum of 
the singular values of X. Keshavan, Montanari, and Oh offer an alternative 
algorithm, dubbed OptSpace, which is based on trimming the incomplete 
matrix to remove so-called “overrepresented” rows and columns whose 
values do not help reveal much about the unknown entries and then 
adjusting the trimmed matrix to minimize the error that is made at the 
entries whose values are known via a gradient descent procedure [link], 
[link]. There are other algorithms as well, and which algorithm to choose is 
really up to the user. For our work, we elected to use the OptSpace 
algorithm, since it just seems to produce better results. 


Simulation Procedure 

This module is part of a collection of modules for a class project on matrix 
completion techniques for the sensor network localization problem done for 
the Fall, 2009 offering of Prof. Baraniuk's ELEC 301 course at Rice 
University. 


Simulation Procedure 


For our project, we applied these new matrix completion techniques to the 
sensor network localization problem. More explicitly, our idea was to take 
an incomplete matrix of distances between sensors and use the OptSpace 
algorithm mentioned previously to fill in the missing entries, whereupon the 
network may be reconstructed using multidimensional scaling methods. 
Because a matrix of Euclidean distances between random points is, in 
general, full rank, it cannot be completed directly; however, the matrix of 
the squares of the distances between the points has a fixed maximum rank 
depending on the dimension of the space in which the points are embedded. 
To see this, suppose that we are given N points 71,..., 2, in IR”, and let Do 
be the NV-by-N matrix of their squared distances; that is, the 27-entry of D» 
is equal to || x; — a; ||? fori, 7 = 1,...,n. Denote the k™ coordinate of zx; 
(k) 


by x”. Because || 2; — a; ||2=|| 2; ||2 —2 (x; e x;) + || a; ||° (where e 
denotes the usual dot product on IR”), we have 
Equation: 
1 pe 1 
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and so Dz may be written as the product of a matrix with n + 2 columns 
and a matrix with n + 2 rows. The rank of Dz may therefore not exceed 
nm + 2. For our particular project, we restricted our attention to sensors 


embedded in a plane (in which case the rank of D2 is at most 4 for any 
number of sensors JV), but this property of the matrix D2 offers a simple 
way to extend our work to higher dimensions. 


To try out our ideas, we designed and executed several different MATLAB 
simulations, each of which proceded according to the following general 
outline: 


1. Generate N = 200 uniformly distributed random points inside the unit 
square [0, 1] x [0,1]. 

2. Form the matrix D of pairwise distances between the points. Add 
noise if necessary. 

3. Form the matrix D2 of the squares of the (possibly noisy) distances 
between the points. 

4. Knock out pairs of distances in D2 according one of two procedures 
(described below) to form the partially observed matrix R. 


5. Complete the matrix R using OptSpace to get Ds: 
6. Form the matrix D, which is the element-wise square-root of Do. 
7. Compare the completed matrix D to the original D by measuring the 


relative Frobenius-norm error e =|| D—D || ,/|| D || p. 
8. Repeat the above steps for 25 trials, and compute the average relative 
Frobenius-norm error at the end. 


We used two different methods for determining which entries in the matrix 
to eliminate, which we call “random" and “realistic” knock-out, 
respectively. By random knock-out, we mean that distance pairs were 
selected at random to be knocked-out according to a fixed probability. In 
constrast, realistic knock-out involves removing all entries of the matrix 
that exceed a certain threshold distance. The idea is that in a realistic 
setting, sensors which are far apart from each other may not be able to 
construct an estimate of the distance between themselves. 


To simulate noise in the trials that required it, we randomly generated 
values from zero-mean Gaussian distributions and added them to the entries 
in the matrix of distances. In order to understand what effect the noise 


amplitude would have on the results, we used five different values for the 
standard deviations of these distributions: 0.01, 0.05, 0.1, 0.2, and 0.5. 


A copy of the MATLAB code we wrote for the simulations is available 
here. The OptSpace code must be downloaded separately and may be found 
at the OptSpace website listed in this project's References module. 


Results, Conclusions, and Future Work 

This module is part of a collection of modules for a class project on matrix 
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Results, Conclusions, and Future Work 


Random Knock-Out Trials 
The results from the simulations for the random knock-out runs are 


displayed in the figures below, which depict the average relative Frobenius- 
norm error over 25 trials versus fraction of unknown entries. 


Ave. F-Norm Error vs. Frac. of Unknown Entries (Random Knock-Out, No Noise) 
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Simulation results for random knock-out trials with no noise. 


Ave. F-Norm Error vs. Frac. of Unknown Entries (Random Knock-Out, Noise) 
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Simulation results for random knock-out trials with noise 
present. 


As these two figures illustrate, the results for the random knock-out trials 
were quite good. As expected, as the fraction of unknown entries becomes 
large, the error eventually becomes severe, while for very low fractions of 
unknown entries, the error is extremely small. What is amazing is that for 
moderate fractions of unknown entries the algorithm still performs 
remarkably well, and its performance doesn't degrade much by the loss of a 
few more entries: the graphs are nearly flat over the range from 0.3 to 0.8! 
As the second figure shows (and as might be imagined), noise only makes 
the error worse; however, the plot also shows that the algorithm is 


reasonably robust to noise in that perturbations of the distance data by small 
amounts of noise don't become magnified into massive errors. 


As an example, consider Figure 3 below, which displays the results of a 
typical no-noise random knock-out run with knock-out probability 0.5. On 
the left is a plot of the sparsity pattern for the incomplete matrix. A blue dot 
represents a known entry, while a blank space represents an unknown one. 
On the right is a plot of what the network looks like after being 
reconstructed using multidimensional scaling. Observe that the red circles 
for the network corresponding to the network generated by the completed 
matrix enclose the blue dots of the original network's structure quite well, 
indicating that the match is very good. 


Incomplete Matrix Sparsity Pattern Reconstructed Network 
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Results from a typical no-noise random knock-out trial with a knock- 
out probability of 0.5. Left: Sparsity pattern for the incomplete matrix. 
Right: Overlay figure demonstrating degree of agreement between the 
original network and the network generated from the completed 
matrix. 


For an illustration of how the results look with noise, see Figure 4 below. 
This figure shows the results of a typical noise-present random knock-out 
run with knock-out probability of 0.5 and noise standard deviation 0.05. 
The agreement in the reconstructed network is not as good as it was for the 
no-noise case, but the points of the reconstructed network are “clustered” in 
the right locations, and some of the prominent features of the original 
network are present in the reconstructed one as well. 


Incomplete Matrix Sparsity Pattern Reconstructed Network 
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Results from a typical noise-present random knock-out trial with a 
knock-out probability of 0.5 and a noise standard deviation of 0.05. 
Left: Sparsity pattern for the incomplete matrix. Right: Overlay figure 
demonstrating degree of agreement between the original network and 
the network generated from the completed matrix. 


Realistic Knock-Out Trials 


The figures below, which show the results for the realistic knock-out trials, 
are similar to those above except that they plot the average relative 


Frobenius-norm error over 25 trials versus maximum radius as opposed to 
fraction of unknown entries. 


Ave. F-Norm Error vs. Max. Radius (Realistic Knock-Out, No Noise) 
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Simulation results for realistic knock-out trials without noise. 


Ave. F-Norm Error vs. Max. Radius (Realistic Knock-Out, Noise) 
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Simulation results for realistic knock-out trials with noise 
present. 


The most salient feature of these graphs is the odd “hump” that appears 
from radius values of about 0.5 to 0.7, even in the no-noise case. Over this 
range, despite the fact that the radius is growing (meaning that more 
pairwise distances are known), the error in the completed matrix is actually 
becoming worse rather than better, which seems to contradict the excellent 
results discussed above for the random knock-out case. At the time of this 
writing, we are still unsure as to why this “hump" appears; however, we 
suspect that it may have something to do with the OptSpace algorithm itself 
because when we run the same experiment using the SVT algorithm of 
Candeés, the hump does not appear, as the figure below shows. (Note that, 
nevertheless, OptSpace tends to produce less error than SVT, even over the 
offending range of radii.) 


Ave. F-Norm Error vs. Max. Radius (Realistic Knock-Out, No Noise} 
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OptSpace performance vs. SVT performance for realistic entry 
knock-out without noise. SVT does not display a “hump," but 
OptSpace generally returns better error values. 


Perhaps more important than the “hump," however, is the fact that the 
scales on the axes of the above graphs alone are enough to demonstrate that 
the performance of the method in the realistic knock-out case is decidedly 
worse than that for the random knock-out case. For example, consider the 
figure below, which shows the results of a typical no-noise, realistick 
knock-out trial with a maximum radius of 1. For this particular trial, over 97 
percent of the pairs are known. The reconstructed network matches the 
original quite well near the “center” of the network, but at the edges, the 
match becomes much worse. This behavior is not exhibited at all by random 
knock-out trials for comparable fractions of unknown entries, as the picture 
at the bottom of the figure illustrates, which was generated from a non- 
noise random knock-out trial in which 90 percent of the pairs were known. 


Incomplete Matrix Sparsity Pattern 
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Reconstructed Network 
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Results from a typical non-noise realistic knock-out trial with a 
maximum radius of 1. Top-Left: Sparsity pattern for the incomplete 
matrix. Top-Right: Overlay figure demonstrating degree of agreement 
between the original network and the network generated from the 
completed matrix. Bottom: Typical results from a non-noise random 
knock-out trial with knock out probability of 0.1 (90% of distance 
pairs are known). 


Shrinking the radius only makes matters worse, as the next figure 
illustrates. The maximum radius here is V2 /2. Around 77 percent of the 


distance pairs are known, and yet the match is terrible. 


Reconstructed Network 
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Results from a typical non-noise realistic knock-out trial with a 
maximum radius of \/2 /2. Left: Sparsity pattern for the incomplete 
matrix. Right: Overlay figure demonstrating degree of agreement 
between the original network and the network generated from the 
completed matrix. 


Adding noise only makes the results even worse. At first glance, this 
behavior appears to be inexplicable; however examining the sparsity 
patterns of the incomplete matrices reveals an interesting fact: the entry 
knock-out in the realistic case is far from being “random!" The sparsity 
patterns for the realistic knock-out matrices reveal clear patterns of lines in 
their knocked-out entries that are not present in those for random knock-out 
cases. This unintended regularity of entry selection violates the assumption 
made in all of the matrix completion literature that the known entries are 
taken from a uniform sampling of the matrix, so it would seem that none of 
the theoretical results that have been derived apply in this case. 


Conclusions 


Our results show that matrix completion presents a viable means of 
approaching the sensor network localization problem under the assumption 
that the known pairs of distances come from a uniform sampling of the 
distance matrix. Under these conditions, matrix completion provides 
excellent network reconstruction and is fairly robust to noise. 
Unfortunately, its performance in the more realistic case in which distance 
information will be excluded or included based on a maximum possible 
distance over which two sensors can communicate leaves much to be 
desired. 


Future Work 


With more time on this project, we would like to have explored the 
following questions further: 


e What is the true origin of the mysterious “hump?" If it really is due to 
OptSpace as the above seems to suggest, is there a way to modify the 
OptSpace algorithm to get it to go away? 

e¢ What is the fundamental reason that the realistic knock-out trials did 
not work? Is there a way to get them to work better? (Perhaps 
something like permuting the distance entries in the matrix around to 
make the sampling pattern apparently more random would do the trick. 
If the permutations are stored somewhere, they can be undone after the 
matrix is completed if necessary.) 

e The experiment worked pretty well in two dimensions, at least for the 
random knock-out case. Will three dimensions show results that are 
any different? 
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